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DESCRIPTION 

IMAGE PROCESSING APPARATUS AND METHOD, INFORMATION 
PROCESSING APPARATU S AND METHOD, RECORDING MEDIUM, AN D 
PROGRAM 



Technical Field 

The present invention relates to an apparatus and a 
method for image processing, an apparatus and a method 
for information processing, a recording medium, and a 
program, and particularly to an apparatus and a method 
for image processing, an apparatus and a method for 
information processing, a recording medium, and a program 
that enable appropriate divided display of a plurality of 
images . 



Background Art 

FIG. 1 shows an example of configuration of a 
conventional image processing system. 

N video cameras 2-1 to 2-N (referred to simply as a 
video camera 2 when the video cameras 2-1 to 2-N do not 
need to be differentiated from each other, and the same 
applies in other cases) disposed at predetermined 
positions with respect to a subject 1 'supply an image 

1 



S03P0906 



obtained as a result of image pickup to an image 
processing apparatus 3. 

The image processing apparatus 3 generates an image 
in which the N images supplied from the video camera 2 
are dividedly displayed on a display 4A of a display 
device 4. The image processing apparatus 3 supplies the 
generated image to the display device 4. The display 
device 4 displays the image from the image processing 
apparatus 3 on the display 4A. 

The content of processing of the image processing 
apparatus 3 differs depending on the number of video 
cameras 2 and a divided display format. 

For example, the content of processing of the image 
processing apparatus 3 differs between a case where nine 
(N = 9) video cameras 2-1 to 2-9 are disposed so as to 
surround a subject 1 as shown in FIG. 2 and nine images 
obtained as a result of image pickup by the video camera 
2 are dividedly displayed as shown in FIG. 3 and a case 
where eight (N = 8) video cameras 2-1 to 2-8 are disposed 
such that scenery in all directions of 360 degrees is a 
subject 1 as shown in FIG. 4 and eight images obtained as 
a result of image pickup by the video camera 2 are 
dividedly displayed as shown in FIG. 5. 

A number attached to a small screen W shown in FIG. 
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3 and FIG. 5 indicates a video camera 2 as a providing 
source of an image to be displayed on the small screen W, 
and is associated with a number succeeding the video 
camera 2. Specifically, for example, an image obtained as 
a result of image pickup by the video camera 2-1 is 
displayed on a small screen Wl, and an image obtained as 
a result of image pickup by the video camera 2-2 is 
displayed on a small screen W2 . 

In addition, in order to properly display each of 
the images obtained as a result of image pickup by the 
video camera 2 on the small screen W, these images need 
to be corrected on the basis of lens distortion of the 
video camera 2 or the like. However, since the lens 
distortion or the like differs between different video 
cameras 2, the image processing performed by the image 
processing apparatus 3 also becomes different accordingly. 

Thus, the image processing apparatus 3 has been a 
dedicated apparatus meeting these conditions. The 
conventional image processing system consequently cannot 
deal readily with a change in the use condition. 

Disclosure of the Invention 

The present invention has been made in view of such 
a situation, and it is an object of the present invention 
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to make it possible to deal readily with a change in the 
use condition. 

According to the present invention, there is 
provided an image processing apparatus including: 
detecting means for detecting coordinates on a small 
screen for displaying a small image including 
predetermined coordinates on a large screen for 
displaying a large image including a plurality of small 
images disposed at predetermined positions, the 
coordinates on the small screen corresponding to the 
predetermined coordinates on the large screen; reading 
means for reading a pixel value of a pixel of the 
predetermined small image, the pixel being located at a 
position corresponding to the coordinates on the small 
screen detected by the detecting means; and outputting 
means for outputting the pixel value read by the reading 
means as a pixel value of a pixel of the large image, the 
pixel being located at a position corresponding to the 
predetermined coordinates on the large screen. 

The image processing apparatus further includes 
storing means for storing a table in which the 
coordinates on the large screen, the coordinates on the 
small screen including the coordinates on the large 
screen, the coordinates on the small screen corresponding 
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to the coordinates on the large screen, and information 
for identifying the small image to be displayed on the 
small screen are associated with each other, wherein the 
detecting means can detect the coordinates on the small 
screen including the predetermined coordinates on the 
large screen, the coordinates on the small screen 
corresponding to the predetermined coordinates, from the 
table; and the reading means can read the pixel value of 
the pixel of the small image identified by the 
information for identifying the small image, the 
information being associated with the predetermined 
coordinates in the table, the pixel being located at the 
position corresponding to the coordinates on the small 
screen detected by the detecting means. 

The small image can be an image corresponding to a 
picked-up image obtained as a result of image pickup by 
an image pickup device. 

According to the present invention, there is 
provided an image processing method including: a 
detecting step for detecting coordinates on a small 
screen for displaying a small image including 
predetermined coordinates on a large screen for 
displaying a large image including a plurality of small 
images disposed at predetermined positions, the 
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coordinates on the small screen corresponding to the 
predetermined coordinates on the large screen; a reading 
step for reading a pixel value of a pixel of the 
predetermined small image, the pixel being located at a 
position corresponding to the coordinates on the small 
screen detected by a process of the detecting step; and 
an outputting step for outputting the pixel value read by 
a process of the reading step as a pixel value of a pixel 
of the large image, the pixel being located at a position 
corresponding to the predetermined coordinates on the 
large screen. 

According to the present invention, there is 
provided a program on a first recording medium, the 
program including:, a detecting control step for 
controlling detecting coordinates on a small screen for 
displaying a small image including predetermined 
coordinates on a large screen for displaying a large 
image including a plurality of small images disposed at 
predetermined positions, the coordinates on the small 
screen corresponding to the predetermined coordinates on 
the large screen; a reading control step for controlling 
reading a pixel value of a pixel of the predetermined 
small image, the pixel being located at a position 
corresponding to the coordinates on the small screen 
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detected by a process of the detecting control step; and 
an outputting control step for controlling outputting the 
pixel value read by a process of the reading control step 
as a pixel value of a pixel of the large image, the pixel 
being located at a position corresponding to the 
predetermined coordinates on the large screen. 

According to the present invention, there is 
provided a first program that makes a computer perform a 
process including: a detecting control step for 
controlling detecting coordinates on a small screen for 
displaying a small image including predetermined 
coordinates on a large screen for displaying a large 
image including a plurality of small images disposed at 
predetermined positions, the coordinates on the small 
screen corresponding to the predetermined coordinates on 
the large screen; a reading control step for controlling 
reading a pixel value of a pixel of the predetermined 
small image, the pixel being located at a position 
corresponding to the coordinates on the small screen 
detected by a process of the detecting control step; and 
an outputting control step for controlling outputting the 
pixel value read by a process of the reading control step 
as a pixel value of a pixel of the large image, the pixel 
being located at a position corresponding to the 
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predetermined coordinates on the large screen. 

The image processing apparatus and method, and the 
first program according to the present invention detect 
coordinates on a small screen for displaying a small 
image including predetermined coordinates on a large 
screen for displaying a large image including a plurality 
of small images disposed at predetermined positions, the 
coordinates on the small screen corresponding to the 
predetermined coordinates on the large screen, read a 
pixel value of a pixel of the predetermined small image, 
the pixel being located at a position corresponding to 
the detected coordinates on the small screen, and output 
the read pixel value as a pixel value of a pixel of the 
large image, the pixel being located at a position 
corresponding to the predetermined coordinates on the 
large screen. 

According to the present invention, there is 
provided an information processing apparatus including: 
first detecting means for detecting coordinates on a 
small screen including coordinates on a large screen, the 
coordinates on the small screen corresponding to the 
coordinates on the large screen; second detecting means 
for detecting information for identifying a small image, 
the information being associated with the coordinates on 
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the large screen; and generating means for generating a 
first table by storing the coordinates on the large 
screen, the coordinates on the small screen detected by 
the first detecting means, and the information for 
identifying the small image detected by the second 
detecting means in association with each other. 

The small image can be an image corresponding to a 
picked-up image obtained as a result of image pickup by 
an image pickup device, and the coordinates on the large 
screen can be associated with information for identifying 
the image pickup device having a corresponding image 
pickup area as the information for identifying the small 
image in each area corresponding to the image pickup area 
of the image pickup device. 

The information processing apparatus further 
includes storing means for storing a second table in 
which coordinates on the small screen of the picked-up 
image for the small screen which image is cut out from 
the picked-up image after being subjected to correction 
on a basis of a condition of an optical system of the 
image pickup device and coordinates on the large screen 
when the picked-up image for the small screen is 
positioned on the predetermined small screen, the 
coordinates on the large screen corresponding to the 
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coordinates on the small screen, are associated with each 
other, wherein the first detecting means can detect 
coordinates on the small screen corresponding to 
coordinates on the large screen from the second table, 
and detect coordinates on the small screen of the picked- 
up image before the correction from the detected 
coordinates on the small screen. 

According to the present invention, there is 
provided an information processing method including: a 
first detecting step for detecting coordinates on a small 
screen including coordinates on a large screen, the 
coordinates on the small screen corresponding to the 
coordinates on the large screen; a second detecting step 
for detecting information for identifying a small image, 
the information being associated with the coordinates on 
the large screen; and a generating step for generating a 
first table by storing the coordinates on the large 
screen, the coordinates on the small screen detected by a 
process of the first detecting step, and the information 
for identifying the small image detected by a process of 
the second detecting step in association with each other. 

According to the present invention, there is 
provided a program on a second recording medium, the 
program including: a first detecting control step for 
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controlling detecting coordinates on a small screen 
including coordinates on a large screen, the coordinates 
on the small screen corresponding to the coordinates on 
the large screen; a second detecting control step for 
controlling detecting information for identifying a small 
image, the information being associated with the 
coordinates on the large screen; and a generating control 
step for controlling generating a first table by storing 
the coordinates on the large screen, the coordinates on 
the small screen detected by a process of the first 
detecting control step, and the information for 
identifying the small image detected by a process of the 
second detecting control step in association with each 
other. 

According to the present invention, there is 
provided a second program including: a first detecting 
control step for controlling detecting coordinates on a 
small screen including coordinates on a large screen, the 
coordinates on the small screen corresponding to the 
coordinates on the large screen; a second detecting 
control step for controlling detecting information for 
identifying a small image, the information being 
associated with the coordinates on the large screen; and 
a generating control step for controlling generating a 
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first table by storing the coordinates on the large 
screen, the coordinates on the small screen detected by a 
process of the first detecting control step, and the 
information for identifying the small image detected by a 
process of the second detecting control step in 
association with each other. 

The information processing apparatus and method, 
and the second program according to the present invention 
detect coordinates on a small screen including 
coordinates on a large screen, the coordinates on the 
small screen corresponding to the coordinates on the 
large screen, detect information for identifying a small 
image, the information being associated with the 
coordinates on the large screen, and generate a first 
table by storing the coordinates on the large screen, the 
detected coordinates on the small screen, and the 
detected information for identifying the small image in 
association with each other. 

Brief Description of the Drawings 

FIG. 1 is a block diagram showing an example of 
configuration of a conventional image processing system. 

FI G.. 2 i s a diagram showing an example of 
arrangement of video cameras in FIG. 1. 
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FIG. 3 is a diagram showing an example of display 
of images obtained as a result of image pickup by the 
video cameras in FIG. 2. 

FIG. 4 is a diagram showing another example of 
arrangement of the video cameras in FIG. 1. 

FIG. 5 is a diagram showing another example of 
display of images obtained as a result of image pickup by 
the video cameras in FIG. 4. 

FIG. 6 is a block diagram showing an example of 
configuration of an image processing system to which the 
present invention is applied. 

FIG. 7 is a diagram showing size of a display of a 
display device in FIG. 6. 

FIG. 8 is a block diagram showing an example of 
configuration of an image processing apparatus in FIG. 6. 

FIG. 9 is a block diagram showing an example of 
configuration of a coordinate conversion table generating 
apparatus in FIG. 6. 

FIG. 10 is a diagram showing another example of 
display of images obtained as a result of image pickup by 
the video cameras in FIG. 2. 

FIG. 11 is a flowchart of assistance in explaining 
image processing of the image processing apparatus in FIG, 
6. 
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FIG. 12 is a diagram showing an example of a 
coordinate conversion table. 

FIG. 13 is a flowchart of assistance in explaining 
operation of the coordinate conversion table generating 
apparatus in FIG. 6 when generating the coordinate 
conversion table. 

FIG. 14 is a diagram showing an example of a 
correspondence table of coordinates on the display and 
camera numbers. 

FIG. 15 is a diagram showing an example of a 
correspondence table of coordinates on the display and 
coordinates on small screens of SD image after correction 

FIG, is a flowchart of assistance in explaining 

a process of generating the correspondence table of FIG. 
15. 

FIG. 17A is a diagram of assistance in explaining a 
process of generating the correspondence table of FIG. 15 

FIG. 17B is a diagram of assistance in explaining a 
process of generating the correspondence table of FIG. 15 

FIG. 17C is a diagram of assistance in explaining a 
process of generating the correspondence table of FIG. 15 

FIG. 18 is a diagram showing an example of 
arrangement of video cameras in FIG. 6. 

FIG. 19 is a diagram showing an example of display 
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of images obtained as a result of image pickup by the 
video cameras in FIG. 18. 

FIG. 20 is a diagram of assistance in explaining 
image pickup areas of the video cameras in FIG. 18. 

FIG. 21 is another diagram of assistance in 
explaining the image pickup areas of the video cameras in 
FIG. 18. 

FIG. 22 is another diagram of assistance in 
explaining the image pickup areas of the video cameras in 
FIG. 18. 

FIG. 23 is another diagram of assistance in 
explaining the image pickup areas of the video cameras in 
FIG. 18. 

FIG. 24 is another diagram of assistance in 
explaining the image pickup areas of the video cameras in 
FIG. 18. 

FIG. 25 is a diagram showing another example of a 
coordinate conversion table. 

FIG. 26 is a flowchart of assistance in explaining 
another operation of the coordinate conversion table 
generating apparatus in FIG. 6 when generating the 
coordinate conversion table. 

FIG. 27 is a diagram showing another example of a 
correspondence table of coordinates on the display and 
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camera numbers. 

FIG i >^S^±s a diagram showing areas on the display 
corresponding to the image pickup areas of the video 
cameras in FIG. 18. 

FIG. 2 9 is another diagram showing areas on the 
display corresponding to the image pickup areas of the 
video cameras in FIG. 18. 

FIG. 30 is another diagram showing areas on the 
display corresponding to the image pickup areas of the 
video cameras in FIG. 18. 

FIG. 31 is a diagram showing another example of a 
correspondence table of coordinates on the display and 
coordinates on small screens of SD images after 
correction . 

FIG. 32 is a flowchart of assistance in explaining 
a process of generating the correspondence table of FIG. 
31. 

FIG. 33 is a diagram showing an external appearance 
of an omnidirectional camera. 

FIG. 34 is a flowchart of assistance in explaining 
another operation of the coordinate conversion table 
generating apparatus in FIG. 6 when generating the 
coordinate conversion table. 

FIG. 35 is a flowchart of assistance in explaining 
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a process of generating a correspondence table of 
coordinates on the display and coordinates on small 
screens of SD images after correction. 

Best Mode for Carrying out the Invention 

FIG. 6 shows an example of configuration of an 
image processing system to which the present invention is 
applied. In the figure, parts corresponding to those in 
FIG. 1 are identified by the same reference numerals. 

N video cameras 2-1 to 2-N disposed at 
predetermined positions with respect to a subject 1 
supply an image (for example an SD (standard definition) 
image) obtained as a result of image pickup to an image 
processing apparatus 11. 

The image processing apparatus 11 generates an 
image (an HD (high definition) image) in which the N SD 
images are dividedly displayed in a predetermined form on 
a display 12A of a display device 12 from the SD images 
supplied from the respective video cameras 2-1 to 2-N 
using a coordinate conversion table Ta generated by a 
coordinate conversion table generating apparatus 13 and 
provided via a memory card 21. The image processing 
apparatus 11 supplies the generated HD image to the 
display device 12. 
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The display device 12 displays the HD image from 
the image processing apparatus 11 on the display 12A. 
Incidentally, the display 12A has a resolution of 1920 X 
1080 pixels as shown in FIG. 7. 

The coordinate conversion table generating 
apparatus 13 generates a coordinate conversion table Ta 
corresponding to use conditions such as the number of 
video cameras 2, arrangement positions of the video 
cameras 2, lens distortion, a display format of the SD 
images, and the like. The coordinate conversion table 
generating apparatus 13 stores the coordinate conversion 
table Ta in the memory card 21, for example, to provide 
the coordinate conversion table Ta to the image 
processing apparatus 11. 

FIG. 8 shows an example of configuration of the 
image processing apparatus 11. 

N decoders 31-1 to 31-N are supplied with an SD 
image from the corresponding video cameras 2-1 to 2-N. 
The decoders 31-1 to 31-N decode the SD image input from 
the video camera 2, and then supply the decoded SD image 
to corresponding field memories 32-1 to 32-N. 

The field memory 32 stores the image data supplied 
from the decoder 31 in field units. 

A converting unit 33 reads the image data stored in 
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the field memories 32-1 to 32-N as required. The 
converting unit 33 generates an HD image in which N 
images are dividedly displayed in a predetermined form on 
the display 12A of the display device 12 from the read 
image data on the basis of the coordinate conversion 
table Ta stored in a storage unit 36. The converting unit 
33 then supplies the HD image to a frame memory 34. The 
frame memory 34 stores the HD image supplied from the 
converting unit 33. 

An encoder 35 reads and encodes image data (image 
data of the HD image) stored in the frame memory 34 as 
required. The encoder 35 supplies image data obtained as 
a result to the display device 12. 

The storage unit 36 reads the coordinate conversion 
table Ta from the memory card 21 loaded in the image 
processing apparatus 11 via an interface 37 , and then 
stores the coordinate conversion table Ta. 

FIG. 9 shows an example of configuration of the 
coordinate conversion table generating apparatus 13. 

A CPU (Central Processing Unit) 41 is connected 
with an input/output interface 46 via a bus 45. When a 
command is input from an input unit 48 including a 
keyboard, a mouse and the like via the input/output 
interface 46, the CPU 41 loads into a RAM (Random Access 
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Memory) 43 a program (for example, a program for 
generating the coordinate conversion table Ta) stored in 
a ROM (Read Only Memory) 42, on a hard disk 44, or on a 
recording medium such as a magnetic disk 61, an optical 
disk 62, a magneto-optical disk 63, a semiconductor 
memory 64 or the like loaded into a drive 50, and the CPU 
41 executes the program. Further, the CPU 41 outputs a 
result of the process to an output unit 47 including an 
LCD (Liquid Crystal Display) and the like via the 
input/output interface 46, for example, as required. 
Incidentally, the program can be provided for a user 
integrally with the coordinate conversion table 
generating apparatus 13 by being stored on the hard disk 
44 or in the ROM 42 in advance, provided as a packaged 
medium such as the magnetic disk 61, the optical disk 62, 
the magneto-optical disk 63, the semiconductor memory 64 
or the like, or provided from a satellite, a network or 
the like to the hard disk 44 via a communication unit 49. 

The CPU 41 stores the generated coordinate 
conversion table Ta in the memory card 21 via the drive 
50. 

Description will next be made of operation of the 
image processing apparatus 11 in a case where nine (N = 
9) video cameras 2-1 to 2-9 are disposed so as to 
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surround a subject 1 at a relatively close range as shown 
in FIG - 2 and images obtained as a result of image pickup 
by the video camera 2 are dividedly displayed as shown in 
FIG. 10 (such a use condition will hereinafter be 
referred to as a first use condition as appropriate) with 
reference to a flowchart of FIG. 11. 

In step SI, the converting unit 33 of the image 
processing apparatus 11 reads a coordinate conversion 
table Ta as shown in FIG. 12 which table is stored in the 
storage unit 36. 

In the coordinate conversion table Ta, coordinates 
on the display 12A, coordinates on a small screen W 
including the coordinates on the display 12A, the 
coordinates on the small screen W corresponding to the 
coordinates on the display 12A, and a camera number 
assigned to the video camera 2 providing an image to be 
displayed on the small screen W are associated with each 
other. In addition, information indicating that black 
color is to be outputted (information represented by 
"black color output" in FIG. 12) is set to predetermined 
coordinates on the display 12A. Incidentally, coordinates 
on the display 12A associated with the information 
indicating that black color is to be outputted are not 
associated with coordinates on the small screen W or a 
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camera number. 

Next, in step S2 , the converting unit 33 selects 
coordinates of one pixel on the display 12A (coordinates 
of one pixel forming an HD image) . 

In step S3, the converting unit 33 determines 
whether or not the information indicating that black 
color is to be outputted is associated with the 
coordinates on the display 12A selected in step S2 in the 
coordinate conversion table Ta. When the converting unit 
33 determines that the information indicating that black 
color is to be outputted is not associated with the 
coordinates on the display 12A, the process proceeds to 
step S4 to detect a camera number associated with the 
selected coordinates on the display 12A from the 
coordinate conversion table Ta. 

Next, in step S5, the converting unit 33 selects 
the field memory 32 corresponding to the video camera 2 
assigned the camera number detected in step S4. 

In step S6, the converting unit 33 detects 
coordinates on the small screen W associated with the 
coordinates on the display 12A selected in step S2 from 
the coordinate conversion table Ta. 

Next, in step S7, the converting unit 33 reads a 
pixel value of a pixel of an SD image stored in the field 
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memory 32 selected in step S5 at a position corresponding 
to the coordinates on the small screen W detected in step 
S6, and stores the pixel value as a pixel value to be 
output at the coordinates on the display 12A selected in 
step S2 in the frame memory 34. 

When the converting unit 33 determines in step S3 
that the information indicating that black color is to be 
outputted is set, the process proceeds to step S8, where 
the converting unit 33 stores a pixel value displayed as 
black color as a pixel value to. be output at the 
coordinates on the display 12A selected in step S2 in the 
frame memory 34. 

For example, when predetermined coordinates within 
a small screen W5 in FIG. 10 are selected in step S2 , the 
selected coordinates are associated with the camera 
number of the video camera 2-5 and are not associated 
with the information indicating that black color is to be 
outputted in the coordinate conversion table Ta, and 
therefore the camera number of the video camera 2-5 is 
detected (step S3 and step S4) . Next, the field memory 
32-5 corresponding to the video camera 2-5 is selected 
(step S5) , and the coordinates on the small screen W 
associated with the selected coordinates on the display 
12A are detected from the coordinate conversion table Ta 
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(step S6) . 

Then the pixel value of the pixel of the SD image 
(step S5) from the video camera 2-5 at the position 
corresponding to the detected coordinates on the small 
screen W (step S6) is stored as the pixel value to be 
output at the selected coordinates on the display 12A 
(step S7) . 

For example, when coordinates of a screen other 
than small screens Wl to W9 in FIG- 10 (a hatched part in 
the figure) are selected in step S2, since the 
information indicating that black color is to be 
outputted is set to such coordinates in the coordinate 
conversion table Ta (step S3), the pixel value displayed 
as black color is stored as the pixel value to be output 
at the selected coordinates on the display 12A (step S8) . 

Returning to FIG. 11 , when the predetermined pixel 
value is stored in the frame memory 34 in step S7 or step 
S8, the process of step S9, where the converting unit 33 
determines whether or not all coordinates on the display 
12A are selected. When the converting unit 33 determines 
that there remain coordinates yet to be selected, the 
process returns to step S2 to select next coordinates on 
the display 12A. 

When the converting unit 33 determines in step S9 
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that all the coordinates are selected, the process is 
ended. 

The coordinate conversion table Ta shown in FIG. 12 
is generated by the coordinate conversion table 
generating apparatus 13 on the basis of the first use 
condition and lens distortion of the video camera 2 as 
later described. Therefore the image processing apparatus 
11 in this system performs the process as described above 
according to the coordinate conversion table Ta, whereby 
the nine SD images supplied from the video camera 2 can 
be dividedly displayed as shown in FIG. 10. 

Description will next be made of operation of the 
coordinate conversion table generating apparatus 13 when 
generating the coordinate conversion table Ta used in the 
above process with reference to a flowchart of FIG. 13. 

In step S21, the CPU 41 of the coordinate 
conversion table generating apparatus 13 selects 
coordinates of one pixel pn the display 12A of the 
display device 12 (coordinates of one pixel forming an HD 
image) (FIG. 10) . 

In step S22, the CPU 41 determines whether or not 
the coordinates on the display 12A selected in step S21 
are within a small screen W (within a screen displaying 
an image corresponding to an SD image obtained as a 
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result of image pickup by the video camera 2). 

Specifically, in this case, the hard disk 44 stores 
a correspondence table Tbl as shown in FIG. 14 in which 
coordinates within the small screens Wl to W9 are 
associated with the camera number of the video camera 2 
providing an image to be displayed at the coordinates, 
and coordinates within the screen other than the small 
screens Wl to W9 (the hatched part in FIG. 10) are 
associated with information indicating the screen other 
than the small screens Wl to W9 (hereinafter referred to 
as outside display area information) . Referring to the 
correspondence table Tbl, the CPU 41 determines whether 
or not the coordinates on the display 12A selected in 
step S21 are associated with the camera number of the 
video camera 2 . 

When the CPU 41 determines in step S22 that the 
coordinates on the display 12A selected in step S21 are 
within a small screen W (when the coordinates selected in 
step S21 are associated with the camera number of the 
video camera 2 in the correspondence table Tbl), the 
process proceeds to step S23, where the CPU 41 detects 
the camera number of the video camera 2 corresponding to 
the coordinates on the display 12A selected in step S21 
from the correspondence table Tbl . 
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Next, in step S24, the CPU 41 detects coordinates 
on the small screen W associated with the coordinates on 
the display 12A selected in step S21 from a 

correspondence table Tb2 stored in the hard disk 44 which 
table indicates correspondences between coordinates on 
the display 12A shown in FIG. 15 and coordinates on the 
small screen W of an image for the small screen W which 
image is cut out from the SD image obtained as a result 
of image pickup by the video camera 2 and corrected on 
the basis of lens distortion of the video camera 2 or the 
like as later described. 

In step S25, the CPU 41 calculates coordinates on 
the small screen W of the SD image before correction (the 
original SD image) as later described from the 
coordinates on the small screen W detected in step S24. 

Next, in step S26, the CPU 41 stores, in the 
coordinate conversion table Ta (FIG. 12), the coordinates 
on the display 12A selected in step S21, the camera 
number detected in step S23, and the coordinates on the 
small screen W calculated in step S25 in association with 
each other. 

When the CPU 41 determines in step S22 that the 
coordinates on the display 12A selected in step S21 are 
not within a small screen W (when the coordinates on the 
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display 12A selected in step S21 are associated with the 
outside display area information in the correspondence 
table Tbl), the process proceeds to step S27. 

In step S27, the CPU 41 stores, in the coordinate 
conversion table Ta (FIG- 12), the coordinates on the 
display 12A selected in step S21 and the information 
indicating that black color is to be outputted in 
association with each other. 

When the predetermined information is stored in the 
coordinate conversion table Ta in step S2 6 or step S27 , 
the process proceeds to step S28, where the CPU 41 
determines whether or not all the coordinates on the 
display 12A are selected on the display 12A. When the CPU 
41 determines that there remain coordinates yet to be 
selected, the process returns to step S21 to select next 
coordinates and perform the subsequent process. 

When- the CPU 41 determines in step S28 that all the 
coordinates are selected, the process is ended. 

Description will next be made of a process of 
generating the correspondence table Tb2 (FIG. 15) used 
when the coordinate conversion table Ta (FIG. 12) is 
generated with reference to a flowchart of FIG. 16. 
Incidentally, suppose in this case that the coordinate 
conversion table generating apparatus 13 generates the 
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correspondence table Tb2 . 

In step S41, the CPU 41 of the coordinate 
conversion table generating apparatus 13 selects one 
camera number from the camera numbers of the video 
cameras 2-1 to 2-9. 

Next, in step S42, assuming an SD image having the 
same size as the SD image obtained as a result of image 
pickup by the video camera 2 (an image having the same 
size as a small screen W in this case) , the CPU 41 
corrects coordinates (Xa, Ya) on the small screen W of 
each pixel forming the SD image (coordinates on the small 
screen W of the SD image before correction) according to 
Equation (1) to thereby calculate coordinates (Xb, Yb) 
(coordinates on the small screen of the SD image after 
correction) . Incidentally, by thus converting the 
coordinates, a blurred peripheral portion of the SD image 
resulting from lens distortion of the video camera 2 can 
be made clear. 

Xb = Xac + (Xa-Xac) (l + k x X r 2 + k 2 X r 4 ) 

Yb = Yac + (Ya-Yac) (l+ki X r 2 +k 2 X r 4 ) 

r = ((Xa-Xac) 2 + (Ya-Yac) 2 ) 2 ...(1) 
In the equation, the coordinates (Xac, Yac) are 

coordinates of a pixel located at a center of the SD 

image. kl and k2 are coefficients determined on the basis 
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of lens distortion of the video camera 2 or the like, r 

is a distance between the pixel at the coordinates (Xa, 

Ya) and a pixel at a center of distortion of the SD image. 

In step S43, the CPU 41 determines an area of the 
SD image obtained as a result of the correction in step 
S42 which area corresponds to the size of the small 
screen W. In step S44, the CPU 41 selects coordinates (Xb, 
Yb) on the small screen W of each pixel of the SD image 
within the determined area. That is, coordinates on the 
small screen W of the SD image after the correction to be 
displayed on the small screen W are selected. 

The SD image obtained as a result of image pickup 
by the video camera 2 (FIG. 17A) is enlarged and deformed 
into a shape of a bobbin as shown by a solid line in FIG. 
17B, for example, by the correction according to Equation 
(1) . From the SD image, the coordinates within the area 
determined in step S43 as shown by a dotted line in FIG. 
17B are selected (FIG. 17C) . Incidentally, coordinates 
(Xai, Yai) in FIG. 17A represent arbitrary coordinates 
(Xa, Ya), and coordinates (Xbi, Ybi) in FIG. 17B and FIG. 
17C represent coordinates (Xb, Yb) obtained as a result 
of correction of the coordinates (Xai, Yai) . 

In step S45, the CPU 41 selects coordinates of one 
pixel from the coordinates on the small screen W of the 
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SD image after the correction, the coordinates on the 
small screen W of the SD image after the correction being 
selected in step S44. In step S46, the CPU 41 converts 
the coordinates into coordinates on the display 12A. 

Specifically, the coordinates on the small screen W 
selected in step S45 are converted into corresponding 
coordinates on the display 12A when the SD image of the 
area determined in step S43 (FIG. 17C) is mapped onto the 
small screen W for displaying the image of the video 
camera 2 having the camera number selected in step S41. 

For example, in a case where the camera number of 
the video camera 2-5 is selected in step S41, the 
coordinates on the small screen W selected in step S45 
are converted into corresponding coordinates on the 
display 12A when the SD image of the area determined in 
step S43 is mapped onto the small screen W5. 

Next, in step S47, the CPU 41 stores, in the 
correspondence table Tb2 (FIG. 15), the coordinates on 
the small screen W of the SD image after the correction, 
the coordinates on the small screen W of the SD image 
after the correction being selected in step S45, and the 
coordinates on the display 12A obtained in step S4 6 in 
association with each other. 

In step S4 8, the CPU 41 determines whether or not 
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all the coordinates selected in step S44 are selected. 
When the CPU 41 determines that there remain coordinates 
yet to be selected, the process returns to step S45 to 
select next coordinates and perform the subsequent 
process . 

When the CPU 41 determines in step S48 that all the 
coordinates are selected, the process proceeds to step 
S49, where the CPU 41 determines whether or not all the 
camera numbers are selected. When the CPU 41 determines 
that there remains a camera number yet to be selected, 
the process returns to step S41 to select the next camera 
number . 

When the CPU 41 determines in step S49 that all the 
camera numbers are selected, the process is ended. 

Thus, a pixel value output at given coordinates Ph 
(not shown) on the display 12A within the small screen W5, 
for example, is a pixel value of a pixel located at 
coordinates (Xai, Yai) on the small screen W (FIG. 17A) 
(step S25 in FIG. 13) of the SD image obtained as a 
result of image pickup by the video camera 2-5 which 
coordinates (Xai, Yai) become coordinates (Xbi, Ybi) on 
the small screen W (FIG. 17B) (step S24 in FIG. 13) 
detected as coordinates corresponding to the coordinates 
Ph on the display 12A as a result of correction in step 

32 



S03P0906 



S42 in FIG. 16. 

That is, without the need for calculating 
coordinates of the input SD image obtained as a result of 
image pickup by the video camera 2 which coordinates 
accommodate lens distortion of the video camera 2 or 
calculating coordinates on the display 12A according to a 
divided display format, the image processing apparatus 11 
can generate an HD image dividedly displaying the SD 
image obtained as a result of image pickup by the video 
camera 2 and subjected to distortion correction by only 
setting pixel values of the SD image in the HD image 
according to the coordinate conversion table Ta as shown 
in FIG. 11. 

Since the coordinates (Xb, Yb) on the display 12A 
of the SD image after the correction in the 
correspondence table Tb2 (FIG. 15) are calculated 
according to Equation (1) as described above, reverse 
correction is performed by optimization corresponding to 
the equation to calculate the coordinates on the small 
screen W of the SD image before the correction in step 
S25 in FIG. 13. 

Description will next be made of operation of the 
image processing apparatus 11 in a case where nine video 
cameras 2-1 to 2-9 are disposed so as to form a plane in 
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a vertical direction with respect to a distant subject 1 
as shown in FIG . 18 and images obtained as a result of 
image pickup by the video camera 2 are dividedly 
displayed as shown in FIG . 19 (such a use condition will 
hereinafter be referred to as a second use condition as 
appropriate) . 

In this example, the video cameras 2 are disposed 
such that image pickup areas of the video cameras 2 
adjacent to each other overlap each other as indicated by 
frames of dotted lines in FIG. 20. For example, the image 
pickup area of the video camera 2-1 (a frame of a solid 
line in FIG. 21) overlaps the image pickup area of the 
video camera 2-2 (a frame of a solid line in FIG. 22), 
the image pickup area of the video camera 2-4 (a frame of 
a solid line in FIG. 23), and the image pickup area of 
the video camera 2-5 (a frame of a solid line in FIG. 24) . 

The image processing apparatus 11 performs the 
process shown in the flowchart of FIG. 11 as in the case 
of the first use form, using a coordinate conversion 
table Ta as shown in FIG. 25 generated by the coordinate 
conversion table generating apparatus 13 on the basis of 
the second use condition and the lens distortion of the 
video camera 2 or the like. That is, although values of 
coordinates and the like handled in the process of the 
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image processing apparatus 11 are changed, the process of 
the image processing apparatus 11 in the case of the 
first use condition and the process of the image 
processing apparatus 11 in the case of the second use 
condition are essentially the same. Therefore it is 
possible to deal readily with a change in the use 
condition. 

Incidentally, unlike the case of the first use 
condition (FIG. 10), an HD image is displayed on the 
whole of the display 12A of the display device 12 in a 
display format of FIG. 19. Therefore, unlike the case of 
the first use condition (FIG. 12), information indicating 
that black color is to be outputted is not set in the 
coordinate conversion table Ta. Thus, in this example, a 
determination of YES is not made in step S3 in the 
flowchart of FIG. 11, and therefore the storing of a 
pixel value for outputting black color is not performed 
in the process of FIG. 8. 

Description will next be made of operation of the 
coordinate conversion table generating apparatus 13 when 
generating the coordinate conversion table Ta (FIG. 25) 
used in this example with reference to a flowchart of FIG. 
26. 

In step S61, the CPU 41 of the coordinate 
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conversion table generating apparatus 13 selects 
coordinates of one pixel on the display 12A of the 
display device 12 (coordinates of one pixel forming ah HD 
image) (FIG. 19) . In step S62, the CPU 41 selects one 
camera number. 

In step S63, the CPU 41 determines whether or not 
the coordinates on the display 12A selected in step S61 
are associated with the camera number selected in step 
S62 in a correspondence table Tel shown in FIG. 27 which 
table shows correspondences between coordinates on the 
display 12A and camera numbers. 

Coordinates on the display 12A in the 
correspondence table Tel shown in FIG. 27 are associated 
with the camera numbers of the video cameras 2 having the 
image pickup areas as shown in FIG. 2 0 corresponding to 
areas as shown in FIG. 28. 

For example, coordinates within the area Ql (FIG. 
29) corresponding to the image pickup area of the video 
camera 2-1 (FIG. 21) on the display 12A are associated 
with the camera number of the video camera 2-1. 
Coordinates within the area Q2 (FIG. 30) corresponding to 
the image pickup area of the video camera 2-2 (FIG. 22) 
on the display 12A are associated with the camera number 
of the video camera 2-2. Incidentally, coordinates on the 
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display 12A belonging to both the area Ql and the area Q2 
(for example coordinates (Xhi, Yhi) in FIG. 29 and FIG. 
30) are associated with the two camera numbers of the 
video cameras 2-1 and 2-2. 

Returning to FIG. 26, when the CPU 41 determines in 
step S63 that the coordinates on the display 12A selected 
in step S61 are associated with the camera number 
selected in step S62, the process proceeds to step S64. 

In step S64, the CPU 41 detects coordinates on a 
small screen W associated with the coordinates on the 
display 12A selected in step S61 and the camera number 
selected in step S62 from a correspondence table Tc2 
stored on the hard disk 44 and shown in FIG. 31 which 
table shows correspondences between coordinates on the 
display 12A, coordinates on the small screen W of an 
image for the small screen W which image is cut out from 
an SD image obtained as a result of image pickup by the 
video camera 2 and corrected on the basis of lens 
distortion of the video camera 2 or the like, and the 
camera number of the video camera 2 having an image 
pickup area corresponding to coordinates on the display 
12A. Incidentally, a method of generating the 
correspondence table Tc2 will be described later. 

Next, in step S65, the CPU 41 calculates 
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coordinates on the small screen W of the SD image before 
the correction from the coordinates on the small screen W 
of the SD image after the correction detected in step S64. 

When the CPU 41 determines in step S63 that the 
coordinates on the display 12A selected in step S61 are 
not associated with the camera number selected in step 
362, or when the coordinates on the small screen W of the 
SD image before the correction are calculated in step S65, 
the process proceeds to step S66, where the CPU 41 
determines whether or not all the camera numbers are 
selected. When the CPU 41 determines that there remains a 
camera number yet to be selected, the process returns to 
step S62 to select the next camera number. 

When the CPU 41 determines in step S66 that all the 
camera numbers are selected, the process proceeds to step 
S67, where the CPU 41 determines whether or not the 
coordinates on the display 12A selected in step S61 are 
associated with a plurality of camera numbers in the 
correspondence table Tel (FIG. 27). When the CPU 41 
determines that the coordinates on the display 12A 
selected in step S61 are associated with a plurality of 
camera numbers, the process proceeds to step S68. 

For example, the coordinates (Xhi, Yhi) on the 
display 12A shown in FIG. 29 and FIG. 30 are associated 
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with both the camera number of the video camera 2-1 and 
the camera number of the video camera 2-3 in the 
correspondence table Tel. Therefore, when the coordinates 
(Xhi, Yhi) are selected in step S61, the process proceeds 
to step S68. 

In step S68, the CPU 41 determines coordinates on 
one small screen W to be set in the coordinate conversion 
table Ta from coordinates on small screens W of a 
plurality of SD images before correction (step S65) which 
coordinates are calculated by performing the process of 
steps S62 to S65 a plurality of times (for example 
coordinates calculated when the camera number of the 
video camera 2-1 is selected in step S62 and coordinates 
calculated when the camera number of the video camera 2-2 
is selected in step S62). 

For example, when the coordinates (Xhi, Yhi) (FIGS. 
29 and 30) are closer to a center of the area Ql than a 
center of the area Q2, coordinates on the display 12A of 
an SD image before correction which coordinates are 
calculated when the camera number of the video camera 2-1 
is selected are selected. 

When the CPU 41 determines in step S67 that the 
coordinates on the display 12A selected in step S61 are 
not associated with a plurality of camera numbers in the 
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correspondence table Tel, or when coordinates on one 
small screen W are determined in step S68, the process 
proceeds to step S69. 

In step S69, the CPU 41 stores the coordinates on 
the display 12A selected in step S61, the coordinates on 
the small screen W calculated in step S65 when the 
coordinates on the display 12A selected in step S61 are 
not associated with a plurality of camera numbers or the 
coordinates on the small screen W determined in step S68 
when the coordinates on the display 12A selected in step 
S61 are associated with a plurality of camera numbers, 
and the camera number selected in step S62 when the 
coordinates on the small screen W are calculated, in 
association with each other in the coordinate conversion 
table Ta (FIG. 25) . 

Next, in step S70, the CPU 41 determines whether or 
not all coordinates on the display 12A are selected. When 
the CPU 41 determines that there remain coordinates yet 
to be selected, the process returns to step S61 to select 
next coordinates and perform the subsequent process. 

When the CPU 41 determines in step S70 that all the 
coordinates are selected, the process is ended. 

Description will next be made of a process of 
generating the correspondence table Tb2 (FIG. 31) used 
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when the coordinate conversion table Ta (FIG. 25) 
corresponding to the second use condition is generated 
with reference to a flowchart of FIG. 32. Incidentally, 
suppose in this case that the coordinate conversion table 
generating apparatus 13 generates the correspondence 
table Tb2. 

In step S81, the CPU 41 of the coordinate 
conversion table generating apparatus 13 selects one 
camera number. 

Next, in step S82, assuming an SD image having the 
same size as the SD image obtained as a result of image 
pickup by the video camera 2, the CPU 41 corrects 
coordinates (Xa, Ya) on the small screen W of each pixel 
forming the SD image (coordinates on the small screen W 
of the SD image before correction) according to Equation 
(1) to thereby calculate coordinates (Xb, Yb) 
(coordinates on the small screen of the SD image after 
correction) . 

In step S83, the CPU 41 determines an area of the 
SD image obtained as a result of the correction in step 
S82 which area corresponds to the size of the small 
screen W. In step S84, the CPU 41 selects coordinates (Xb, 
Yb) on the small screen W of each pixel of the SD image 
within the determined area. That is, coordinates on the 
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small screen W of the SD image after the correction to be 
displayed on the small screen W are selected. 

Next, in step S85, the CPU 41 selects coordinates 
of one pixel from the coordinates on the small screen W 
of the SD image after the correction, the coordinates on 
the small screen W of the SD image after the correction 
being selected in step S84. In step S86, the CPU 41 
converts the coordinates into coordinates on the display 
12A. 

Specifically, the coordinates on the small screen W 
selected in step S85 are converted into corresponding 
coordinates on the display 12A when the SD image of the 
area determined in step S83 is mapped onto the small 
screen W for displaying the image of the video camera 2 
having the camera number selected in step S81. 

For example, in a case where the camera number of 
the video camera 2-5 is selected in step S81, the 
coordinates on the small screen W selected in step S85 
are converted into corresponding coordinates on the 
display 12A when the SD image of the area determined in 
step S83 is mapped onto the small screen W5 (FIG. 19) . 

Next, in step S87, the CPU 41 stores the camera 
number selected in step S81, the coordinates on the small 
screen W of the SD image after the correction, the 
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coordinates on the small screen W of the SD image after 
the correction being selected in step S85, and the 
coordinates on the display 12A obtained in step S8 6 in 
association with each other in the correspondence table 
Tc2 as shown in FIG. 31. 

In step S88, the CPU 41 determines whether or not 
all the coordinates selected in step S84 are selected. 
When the CPU 41 determines that there remain coordinates 
yet to be selected, the process returns to step S85 to 
select next coordinates. 

When the CPU 41 determines in step S8 8 that all the 
coordinates are selected, the process proceeds to step 
S8 9, where the CPU 41 determines whether or not all the 
camera numbers are selected. When the CPU 41 determines 
that there remains a camera number yet to be selected, 
the process returns to step S81 to select the next camera 
number . 

When the CPU 41 determines in step S8 9 that all the 
camera numbers are selected, the process is ended. 

Description will next be made of operation of the 
image processing apparatus 11 in a case where eight (N = 
8) video cameras 2-1 to 2-8 are disposed such that 
scenery in all directions of 360 degrees is a subject 1 
as shown in FIG. 4 and images obtained as a result of 
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image pickup by the video camera 2 are dividedly 
displayed as shown in FIG. 5 (such a use condition will 
hereinafter be referred to as a third use condition as 
appropriate) . 

Incidentally, the eight video cameras 2 shown in 
FIG. 4 are in practice arranged at a predetermined angle 
in an omnidirectional camera 101 as shown in FIG. 33 such 
that centers of projection of the video cameras 2 
coincide near a center of the omnidirectional camera 101 
and directions of lines of sight of the video cameras 2 
are in one horizontal plane. 

Plane mirrors 110 are disposed in the directions of 
the lines of sight of the video cameras 2. 

That is, the video cameras 2 pick up an image of 
surrounding scenery reflected by the corresponding plane 
mirrors 110, whereby the omnidirectional camera 101 as a 
whole can pick up an image. of the scenery in 360 degrees 
in a horizontal direction. 

The image processing apparatus 11 performs the 
process shown in the flowchart of FIG. 11 as in the case 
of the first use condition and the second use condition, 
using a coordinate conversion table Ta generated by the 
coordinate conversion table generating apparatus 13 on 
the basis of the third use condition and the lens 
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distortion of the video camera 2 or the like. As a result, 
eight images obtained as a result of image pickup by the 
video camera 2 can be dividedly displayed as shown in FIG. 
5. 

Description will next be made of operation of the 
coordinate conversion table generating apparatus 13 in 
generating the coordinate conversion table Ta used in 
this example with reference to a flowchart of FIG . 34. 

A process performed in steps S101 to S104 is the 
same as in steps S61 to S64 in FIG. 26, and therefore 
description thereof will be omitted. 

In step S105, the CPU 41 optimizes coordinates on a 
small screen W of an SD image after correction detected 
in step S104 and performs a process reverse to a mirror 
inversion process (a reverse mirror inversion process) to 
thereby calculate coordinates on the small screen W of 
the SD image before the correction. 

A process performed in steps S106 to S110 is the 
same as in steps S66 to S70 in FIG. 26, and therefore 
description thereof will be omitted. 

Description will next be made of a process of 
generating a correspondence table used in step S104 which 
table shows correspondences between coordinates on the 
display 12A and coordinates on the small screen W of the 
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SD image after the correction with reference to a 
flowchart of FIG. 35. Incidentally, suppose in this case 
that the coordinate conversion table generating apparatus 
13 generates the correspondence table. 

In step S121, the CPU 41 of the coordinate 
conversion table generating apparatus 13 selects one 
camera number. 

Next, in step S122, assuming an SD image having the 
same size as the SD image obtained as a result of image 
pickup by the video camera 2, the CPU 41 corrects 
coordinates (Xa, Ya) of each pixel forming the SD image 
according to Equation (1) to thereby calculate 
coordinates (Xb, Yb) . Also, the CPU 41 performs the 
mirror inversion process. 

A process performed in steps S123 to S129 is the 
same as in steps S83 to S89 in FIG. 32, and therefore 
description thereof will be omitted. 

The series of processes described above can be 
implemented not only by hardware but also by software. 
When the series of processes is to be implemented by 
software, a program forming the software is installed 
onto a computer, and the computer executes the program, 
whereby the image processing apparatus 11 and the 
coordinate conversion table generating apparatus 13 
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described above are functionally implemented. 

It is to be noted that in the present specification, 
the steps describing the program provided by a recording 
medium include not only processes carried out in time 
series in the described order, but also processes carried 
out in parallel or individually and not necessarily in 
time series. 

Also, in the present specification, a system refers 
to an apparatus as a whole formed by a plurality of 
apparatus . 

Industrial Applicability 

According to the first present invention, it is 
possible to make divided display of small screens easily. 

According to the second present invention, it is 
possible to make divided display of small screens easily 
even when a use condition is changed. 
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